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ABSTRACT 

This short paper describes an ongoing NSF-funded project on enhancing science and engineering education using the 
latest technology. More specifically, the project aims at developing an interactive learning system with Microsoft 
Kinect™ and UnitySD game engine. This system promotes active, rather than passive, learning by employing embodied 
cognition with interactive visual/spatial information, in which human movements could impact a lasting effect on both 
the short-term episodic and long-term memories of students. Two groups of Biomedical Engineering students at The 
University of Akron participated in a study of the four educational modules (Cell, DNA, Immune System, and Gene 
Therapy) and were tested after the conclusion of each educational session. The preliminary results show a trend toward 
better performance with the system compared to traditional instruction in second-year college students. However, more 
studies with a larger group and with a younger population, such as K-12, need to be evaluated since these students have a 
less developed visuo-spatial sketchpad relative to the sophomores in college. 


KEYWORDS 

Interactive Learning, Visuo-Spatial Sketchpad, Natural User Interface, Embodied Cognition. 


1. INTRODUCTION 

When teaching educational contents that are inherently graphical in nature, research has shown that learning 
is more efficient when stimulating the visual component of the brain (the visuo-spatial sketchpad) as 
compared to the language -based approaches [1]. In the working memory model proposed by Baddeley and 
Hitch [2], the central executive controls the flow of information from and to the phonological loop (verbal 
buffer), the visuo-spatial sketchpad (visual buffer), and the episodic buffer (where context is added to short- 
term-memory codes for storage in long-term memory). We know that both verbal and visual codes are stored 
in short-term memory because we can use these two types of codes to interfere with working memory 
performance [2]. Because of the “video game effect,” there is accumulating evidence that individuals can use 
visual codes to help supplement verbal-code information in short-term or working memory. 

Furthermore, there is evidence that humans use “embodied cognition” to improve performance [3-5], 

That is, we encode the state of the body and use it as a “somatic marker” (or contextual cue) when storing 
working-memory information in long-term memory [3-6]. Using these assumptions, it is predicted that 
visual codes and motoric feedback from movements should improve the educational outcomes in learning 
science and engineering information. 
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Figure 1. Components of Baddeley and Hitch’s updated concept of working memory. 

The goal of this project is to refocus science and engineering education by introducing a new technology 
that is integrated with interactive, educational content that engages, inspires, and stimulates the visual center 
of the brain. To accomplish these goals, a new curricula has be designed and implemented that engages 
students and interactively introduce visual information using the Microsoft Kinect™ device [7], This device 
will be coupled to a computer system and will allow students, as an example, to actively explore a cell’s 
anatomy and its functions using an interactive spatial interface instead of simply showing pictures with 
verbal descriptions. This approach also promotes active, rather than passive, learning by employing 
embodied cognition, where human movements could impact a lasting effect on both the short-term episodic 
and long-term memories of students [3-6], In our approach, both the visual/spatial information and the 
motoric interactions can be stored with a memory trace of the learning event (e.g., structures and functions of 
an eukaryotic cell and its organelles). 

We hypothesize that introducing graphical information that is designed specifically to simulate the visuo- 
spatical sketchpad with gestures will positively impact the students’ learning and have a lasting effect on 
cognition and episodic memory [8-9]. Since our educational content is also designed similar to a game 
interface, we will also enhance the emotional states of the students which is associated with potentially better 
episodic memory performance for remembering events [9] as demonstrated by applications of Damasio’s 
somatic marker hypothesis [3-5], Thus, encoding the visual information (in visuo-spatial sketchpad), along 
with integrated gestures, should result in more accurate retrieval of the contextual information than will lead 
to more efficient learning. 


2. SYSTEM DESIGN 

Human gesture recognition is an important part of advancement in human -computer interaction [10]. Recent 
development in natural user interface (NUI) is exemplified by gesture-based interactions. Natural user 
interfaces are user-centric and allow people to interact with the computer in ways that are easier to learn. By 
eliminating the mediator of a keyboard or mouse between the user and the computer, gesture -based 
interactions enable the user to directly interact with the visual content on the computer. Furthermore, if the 
users are represented as an avatar, they will gain the sense of embodiment and be immersed into the 
programmed contents. 

Kinect™ device is the latest natural user interface that provides real-time gesture recognition [11]. In 
addition to a RGB color camera, it has a depth camera, which consists of an infrared projector and an infrared 
sensor [12-14]. Microsoft® has developed an algorithm that utilizes these cameras to recognize human poses 
[15], Furthermore, the depth information enables the system to capture gesture motions in three dimensions; 
therefore, Kinect™ recognizes left-right, up-down, and front-back motions [15]. 

In this project, we use Kinect™ to capture the motion of students and track their gestures to allow them 
visually interact with the education contents. We developed a set of educational contents for Biomedical 
Engineering and created virtual 3D objects for four learning modules: Cell, DNA, Immune System and Gene 
Therapy. Those virtual objects are then imported into a game engine (Unity3D [16]) for animation and 
display. User movements captured by Kinect™ are sent to the game engine through the application- 
programming interface (API) provided by (OpenNI [17] and Zigfu [18]). The students will be able to rotate 
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the virtual objects, for example, the entire cell or its organelles. The students will also be able to zoom onto 
nano-sized molecules such as DNA, ribosomes, and cellular receptors. They will be able to study not only 
their structures but also their functions through interactive animation. They will have the control to 
translocate from outside of the cell into various organelles within the cells. This “first person” perspective 
will use embodied cognition and should import the relevant contextual information more efficiently into the 
student’s long-term memory. These and other features will be linked to the student’s gestures that are 
captured by the Kinect™ system. Since these gestures are intuitive, the leaning curve to use the proposed 
educational modules and the hardware will be minor. 

Figure 2 depicts the architecture of our Kinect™ based system and figure 3 demonstrates the system in 
action. 



Figure 2. The architecture of the Kinect™ based learning system. 


Figure 3. The system in action. 


3. EXPERIMENT DESIGN 

An experimental goal of the project is to compare the learning outcomes for seniors and sophomores 
undergraduate students using traditional verses technology-enhanced (Kinect™) teaching methods in 
biomedical engineering courses. The rationale for this preliminary experimental design is to compare 
populations of students with fully developed visuo-spatial sketchpad (seniors) to a group in which this 
component of their brains is still under development (sophomores). Thus, seniors are more efficient in 
learning by visual means. The learning and understanding of educational materials related to each module 
will be tested. The learning and understanding of educational materials related to each module will be tested. 
The experimental design is given in Table 1. Students will be exposed to both the traditional techniques 
(through modules 2 and 3) and enhanced teaching using the Kinect™ system (through modules 1 and 4). The 
contents for Modules 1-3 are focused upon biological science. In contrast, module 4 is focused upon 
biomedical engineering. This experimental design will allow for intragroup and intergroup comparisons. In 
this way, all students will be exposed to our Kinect™ system. After each session, students were tested. The 
results for the traditional and technology -enhanced teaching methods were averaged and compared using 
paired t-test. 

Table 1 . Experiment groups tested using traditional verses enhance teaching methods. 



Seniors 

Sophomores 

Module 1 - Cell 

Kinect™ 

Kinect™ 

Module 2 - DNA 

Traditional 

Traditional 

Module 3 - Immune System 

Traditional 

Traditional 

Module 4 - Gene Therapy 

Kinect™ 

Kinect™ 
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4. PRELIMINARY RESULTS AND FUTURE WORK 

Two groups of Biomedical Engineering students at the University of Akron participated in study of the four 
educational modules (Cell, DNA, Immune System, and Gene Therapy) and tested after the conclusion of 
each educational session. We expect that seniors using the Kinect™ system will not perform better with 
technology-enhanced teaching method, and the preliminary results show that no differences for seniors using 
both the Kinect™ and traditional approaches (Figure 3, p = 0.92). For sophomores using the Kinect™ 
system, their performance in learning is better with the technology-enhanced method, but the results were not 
statically different (p = 0.49) from the traditional method. However, a correlation exists between the 
Kinect™ and the traditional method for this group (p = 0.034). With n = 6 and 5, respectively, for each 
group, we demonstrated the potential utility of the Kinect™ system’s ability to enhance teaching graphical 
information. However, more studies with a larger group and with a younger population, such as K-12, need 
to be evaluated since these students have a less developed visuo-spatial sketchpad relative to the sophomores 
in college. 



Figure 4. Results of the Kinect™ and the traditional teaching methods for seniors and sophomores. 
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